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The  Software  Supportability  Qualitative  Assessment  Methodology  is  a  five 
volume  referenci;'  set  that  provides  measures  to  aid  in  the  support  of  information  systems. 
These  manuals  are  aimed  at  improving  the  support  process  by  more  accurately  assessing  the 
capabilities  of  support  organizations,  quantitatively  measuring  the  support_jility  of  fielded 
systems  and  evaluating  the  operational  readiness  of  fielded  systems. 

Volume  I,  Developing  Quality  Measures  for  Information  Systems  Support,  describes  the 
three  measures  aloug  with  the  model  of  information  system  support  that  the  measures  are 
designed  to  satisfy.  This  is  the  main  volume  of  the  set  and  should  be  consulted  before 
implementing  the  measures  described  in  more  detail  in  the  other  volumes. 

Volume  II,  The  Review  of  Metrics  for  Developing  an  Information  Systems  Support  Mea¬ 
surement  Framework,  provides  a  survey  and  evaluation  of  current  metrics  in  terrcs  of  in¬ 
formation  systems  support.  Specifically,  three  classes  of  metrics  are  reviewed:  software 
product  metrics,  life  cycle  process  metrics,  and  process  management  metrics. 

Volume  III,  Implementing  the  Software  Supportability  Measure,  provides  instructions  for 
collecting  data  for  the  measure,  compiling  the  measure  by  evaluating  ths  data,  and  inter¬ 
preting  the  final  result.  The  volume  also  contains  guidelines  for  improving  the  supportabilty 
of  an  information  system  based  on  its  evaluation.  Specifically,  the  volume  contains  resource 
estimations  for  compiling  and  evaluating  the  measure,  questionnaires  for  collecting  the  re¬ 
quired  data  and  step-by-step  instructions  for  measuring  the  supportability  of  an  information 
system. 

Volume  IV,  Implementing  the  Support  Organization  Assessment  Measure,  provides  in¬ 
structions  for  collecting  data  for  the  assessment,  conducting  the  assessment,  and  interpret¬ 
ing  the  final  result.  The  volume  also  contains  guidelines  for  improving  the  capabilities  of 
a  support  organization  based  on  its  evaluation.  Specifically,  the  volume  contains  resource 
estimations  for  conducting  and  evaluating  the  assessment,  questionnaires  for  collecting  the 
required  data  and  step-by-step  instructions  for  measuring  the  capabilities  of  a  support  or¬ 
ganization. 

Volume  V,  Implementing  the  Operational  Readiness  Measure,  provides  instructions  for 
collecting  data  foi  the  measure,  compiling  the  meaf.ure  tv  e'-'aluaiing  the  data,  and  inter¬ 
preting  the  final  result.  The  volume  also  contains  guidelines  for  improving  the  operational 
readiness  of  an  information  system  based  on  its  evaluation.  Specifically,  the  volume  contains 
resource  estimations  for  compiling  and  evaluating  the  measure,  questionnaires  for  collecting 
the  required  data  and  step-by-step  instructions  for  measuring  the  operational  readiness  of 
an  .iiformation  system. 
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1  Introduction 


The  support  organization  assessment  measure  is  focused  on  defining  the  important  positive 
factors  that  characterize  an  effective  support  organization.  The  objective  of  the  measure  is 
to  position  an  organization  in  one  of  the  five  levels  of  support  process  maturity  based  on 
that  organization’s  recognition  and  practice  of  these  factors.  The  measure  can  also  highlight 
problem  areas  where  improvements  are  needed. 

This  method  is  patterned  after  the  assessment  method  developed  by  the  Software  £np-  I 

neering  Institute  (SEI)  at  Carnegie  Mellon  University.  SEI  measures  the  quality  of  software 
systems  development  by  using  a  questionnaire  to  elicit  information  about  an  organization’s 
knowledge  and  practice  of  software  development  factors.  Each  question  on  this  quesvionnaire 
is  weighted  according  to  its  position  along  a  five-point  maturity  scale.  The  questionnaire  is 
filled  out  by  members  of  the  development  organization  in  conjunction  with  representatives 
of  SEI.  The  tinswers  are  evaluated  and  the  organization  is  assigned  a  maturity  score  be-  * 

tween  one  and  five  that  correspond  to  five  levels  of  software  development  maturity:  initial, 
repeatable,  defined,  managed,  and  optimized. 

The  SEI  Assessment  is  a  measure  of  an  organization’s  maturity  with  respect  to  the 
software  development  process.  This  organizational  assessment  with  respect  to  the  support  ^ 

process  is  different  from  SEI’s  assessment  of  software  development  because  the  software 
support  process  is  different  than  the  software  development  process.  The  factors  that  deter¬ 
mine  a  mature  software  development  organization  are  not  necessarily  the  same  factors  that 
determine  a  mature  software  support  organization.  Many  factors  are  sbau’ed  by  the  two 
processes  but  the  weights  of  the  common  factors  wiU  be  different  depending  upon  whether 
a  maturity  measurement  is  desired  or  whether  a  measurement  of  software  development  is  ^ 

desired. 

Thus,  the  assesjment  measure  involves  having  a  representative  or  representatives  of 
the  support  organization  answer  questions  that  indicate  that  organization’s  awareness  and 
practice  of  the  software  support  factors  with  respect  to  the  software  support  process.  Each 
question  is  categorized  as  a  Level  1,  2,  3,  4,  or  5  question.  Each  question  requires  a  “yes” 
or  “no”  response.  The  answers  are  tallied  and  an  evaluation  is  made  by  determining  the 
percentage  of  “yes”  answers  that  are  made  w'ith  respect  to  the  questions  in  each  level. 

A  Level  1  organization  will  answer  “yes”  to  80%  of  the  Level  1  questions.  A  Level  2 
organization  will  answer  “yes”  to  80%  of  all  Level  1  and  Level  2  questions;  a  Level  3 
organization  will  answer  “yes”  to  80%  of  all  Level  1,  2  and  3  questions;  and  so  on.  » 

The  questionnaire  that  we  have  developed  for  use  for  the  organizational  assessment  is 
in  Appendix  C.  The  steps  involved  in  developing  this  questionnaire  axe: 

A.  Collect  support  organization  assessment  factors  from  the  organizational  perspective.  ^ 

2.  Categorize  these  factors. 

3.  Devise  “yes”  or  “no”  questions  that  elicit  respondent’s  knowledge  of  or  adherence  to 
these  factors. 

4.  Weigh  the  questions  with  respect  to  software  support  maturity  levels.  » 

5.  Place  these  questions  on  a  Software  Support  Maturity  Matrix. 
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Appendix  D  shows  the  matrix  with  maturity  levels  across  the  chart  and  organizational 
support  factors  on  the  vertical  axis.  Each  cell  of  the  matrix  lists  those  question  numbers 
that  elicit  information  pertaining  to  an  organization’s  awareness  and  practice  of  a  factor 
of  software  support  at  a  certain  level  of  maturity.  For  instance,  question  1.1.8,  “Does  the 
Software  Maintenance  Quality  Assurance  function  have  a  management  reporting  channel 
separate  from  the  software  project  management?”,  is  the  eighth  question  dealing  with  the 
factor  of  organizational  structure.  It  is  a  level  3  question  in  that,  by  answering  “yes”  to 
this  question,  the  organization  has  used  the  organizational  structure  as  a  method  to  ensure 
that  those  responsible  for  the  quality  of  software  support  are  not  controlled  by  those  people 
that  are  tasked  with  project  completion.  The  chart  (Appendix  D)  shows  the  distribution 
of  questions  that  are  used  to  place  an  organization  at  a  particular  level. 

As  previously  noted,  the  overall  maturity  level  of  an  organization  is  based  upon  the 
percentage  of  affirmative  answers  to  each  question  at  each  level  of  maturity  for  all  categories 
of  questions.  However,  an  organization  may  be  a  Level  3  organization  with  respect  to 
organizationzd  issaes,  at  Level  2  for  software  support  process  and  personnel  factors,  and  at 
Level  1  with  respect  to  factors  pertaining  to  tools  and  technology.  The  overall  assessment 
may  result  in  a  Level  2  score.  This  suggests  that  the  organization  would  want  to  become 
more  aware  of  tools  and  technologies  that  may  be  applicable  to  the  support  process  of  that 
organization  in  order  to  provide  better  software  support. 


2  Questionnaire  Usage  Guide 

» 

The  primary  goal  of  the  questionnaire  is  to  help  a  support  organization  evaluate  its  sup¬ 
port  rapabilities..  In  combination  with  this  evaluation,  the  responses  to  the  questions  will 
suggest  factors  that  will  help  the  organization  improve  the  qi’.ality  of  the  support  that  the 
organization  provides. 

• 

Who  Should  Fill  It  Out 

The  intention  of  the  questionnaire  is  that  it  is  a  self-asse.ssment  rather  than  an  assessment 

performed  by  an  outside  agency.  The  specific  person  most  appropriate  for  filling  out  this 

questionnaire  is  the  manager  or  director  of  the  support  organization  tasked  with  supporting  * 

a  portfolio  of  software  rpplications.  The  idea  of  the  questionnaire  is  not  to  point  fingers  at 

any  one  organization,  but  to  know  for  oneself  as  to  how  one’s  own  performance  is.  This  the 

reason  why  this  questionnaire  needs  to  be  filled  by  the  manager  rather  than  anyone  else. 

The  manager  can  get  help  from  users  and  software  developers  and  some  selected  stciff  to  aid 

in  answering  these  questions  truthfully.  If  used  as  specified,  the  questionnaire  can  uncover  » 

problem  aresis  in  addition  to  identifying  the  level  at  which  the  organization  is. 

Material 

Four  items  comprise  the  material  required  to  perform  the  assessment:  the  questionnaire,  ^ 

the  answer  sheet,  the  evaluation  guide  and  the  g’ossary. 

The  questionnaire  contains  a  series  of  questions  that  are  answered  “yes”  or  “no.”  The 


3 


questions  are  categorized  according  to  issues  pertaining  to  management  of  the  organization, 
software  process,  technology,  and  personnel.  Each  question  concerns  an  important  factor 
of  the  support  organization  assessment  measure  that  is  related  to  category  in  which  it  is 
contained.  Many  of  the  questions  may  seem  similar.  However,  in  order  to  properly  answer 
a  question  the  purpose  of  that  question  must  be  viewed  in  the  perspective  of  the  category 
in  which  it  is  located. 

The  answer  sheet  is  a  form  used  to  mark  the  answers  to  the  questions.  It  is  also  used  ais  a 
worksheet  to  determine  the  level  at  which  the  organization  is  operating  (ad-hoc,  repeatable, 
methods,  control,  or  optimal). 

Because  of  differences  in  terminology  cind  differing  software  support  orgcinizations  it  is 
necessary  to  include  a  glossary  of  terms  that  help  in  understanding  the  intention  of  some  of 
the  questions  on  the  questionnaire.  Because  of  the  nature  of  the  “yes”  and  “no”  questions,  it 
is  most  important  that  the  person  filling  out  the  questionnaire  understands  the  terminology 
used  in  the  questions  and  the  intention  of  each  individual  question.  As  mentioned  earlier, 
questions  may  seem  very  similar.  Persons  filling  out  this  questionnaire  should  first  ascertain 
that  they  are  viewing  questions  with  respect  to  the  category  in  which  they  are  contained. 
They  should  also  freely  consult  the  glossary  for  any  terms  that  they  feel  are  ambiguous  or 
unclejir. 

The  evaluation  guide  is  a  short  step  by  step  procedure  for  determining  the  overall  mear 
sure  of  the  quality  of  software  support  supplied  by  the  organization.  This  guide  is  used  with 
the  answer  sheet  to  determine  that  level  at  which  an  organization  supports  the  software 
applications  in  its  portfolio. 

Scoring  Procedure 

In  order  to  rank  support  organizations,  five  maturity  levels  have  been  determined.  These 
levels  represent  evolving  stages  for  software  support  organizations.  An  organization  at  the 
most  rudimentary  stage  of  software  support  would  be  at  level  1,  the  most  sophisticated 
as  predicted  by  researchers  would  be  at  level  5,  and  the  remaining  maturity  levels  would 
indicate  different  degrees  of  evolutionary  growth  eind  capability.  Ranking  an  organization 
consists  of  determining  an  overall  maturity  level  based  upon  the  organization’s  knowledge 
of  and  adherence  to  the  factors  associated  with  the  software  support  process.  For  example, 
a  level  3  organization  would  know  and  adhere  to  almost  all  of  the  factors  associated  with 
levels  1,  2  and  3. 

The  questions  are  designed  to  allow  for  easy  scoring  of  individual  questions.  In  order  to 
determine  the  area-wise  ranking  and  corresponding  level,  a  procedure  requiring  successive 
qualification  at  each  level  is  used.  This  is  as  follows: 

1.  Determine  the  percentage  of  affirmative  answers  to  all  Level  1  questions,  and  if  this 
is  at  ieast  80%  the  organization  has  qualified  for  assessment  for  level  2,  or  else  it  is  at 
level  1. 

2.  Determine  the  percentage  of  affirmative  answers  to  all  Level  2  questions,  and  if  this 
is  at  least  80%,  the  organization  has  qualified  for  assessment  for  level  3,  or  else  it  is 
at  level  2. 
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3.  Determine  the  percentage  of  affirmative  answers  to  all  Level  3  questions,  and  if  this 
is  at  least  80%,  the  organization  has  qualified  for  assessment 

for  level  4,  or  else  it  is  at  level  3. 

4.  Determine  the  percentage  of  affirmative  answers  to  all  Level  4  questions,  and  if  this 
is  at  least  80%,  the  organization  is  at  level  5,  or  else  it  is  at  level  4. 

A  classification  for  overall  maturity  may  be  obtained  by  following  the  same  procedure 
outlined  above  with  the  entire  set  of  questions.  This  overall  classification  helps  to  provide 
a  complete  picture  of  the  support  capability.  Overall  quality  software  support  requires  a 
balamce  of  maturity  in  all  four  categories  of  support  organization  assessment  factors  (see 
Figure  1).  However  area- wise  classification  helps  highlight  specific  strengths  and  weaknesses 
of  the  organization.  The  implications  of  the  different  levels  are  as  per  the  descriptions  in 
the  previous  section  discussing  Levels  of  Software  Supportability. 


3  Suggestions  for  Process  Improvements 

The  levels  of  assessment  of  the  support  organization  provide  a  spectrum  of  maturity  levels. 
As  such,  a  Level  1  organization  cannot  become  a  Level  5  organization  without  first  becoming 
a  Level  2,  3,  and  4  organization. 

To  grow  from  an  ad-hoc,  fire-fighting  level  of  Level  1  to  Level  2  an  organization  must 
have  procedures  which  allow  for  the  collection,  evaluation,  categorization,  and  prioritization 
of  software  problems  and  planning  mechanisms  for  determining  the  schedule  of  the  fixes. 

Level  2  organizations  can  be  counted  on  to  provide  accurate  estimates  of  problem  fixes 
for  those  systems  that  they  support  but  cannot  be  depended  upon  for  any  new  systems 
that  they  may  have  assigned  to  them.  In  order  for  a  Level  2  organization  to  achieve  Level  3 
capabilities,  it  must  have  enough  knowledge  of  the  software  support  process  itself  that  it  has 
defined  the  methods  of  software  support.  This  usually  involves  have  full-time  people  devoted 
to  supporting  the  software  support  process  instead  of  only  providing  software  support.  The 
steps,  methods  and  procedures  used  by  the  organization  need  to  be  documented. 

To  reach  Level  4,  a  Level  3  organization  must  actually  provide  measurements  to  indicate 
that  the  steps,  methods  and  procedures  that  it  has  documented  at  Level  3  are  actually 
followed.  Minimum  measures  are  set  and  actual  costs  and  benefits  can  be  quantified. 

At  Level  5,  the  costs  and  benefits  are  quantified,  recorded,  and  compared  with  past 
performance  to  determine  which  policies,  procedures,  and  resources  are  best  used  in  par¬ 
ticular  circumstances.  The  optimal  software  support  organization  completely  understands 
the  support  process  and  has  efficient  policies  and  procedures  in  place  to  effectively  man¬ 
age  the  resources  available  to  perform  the  corrective,  adaptive,  and  perfective  maintenance 
requirements  of  the  portfolio  of  application  systems  that  it  has  to  support.. 

One  organization  may  perform  at  different  maturity  levels  among  the  separate  categories 
of  factors.  For  example,  a  hypothetical  organization  may  be  a  Level  3  organization  with 
respect  to  organizational  issues,  at  Level  2  for  software  support  process  and  personnel 
factors,  and  at  Level  1  with  respect  to  factors  pertaining  to  tools  and  technology  (see 
Figure  2).  The  overall  assessment  may  result  in  a  Level  2  score.  This  suggests  that  the 
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organization  needs  to  become  more  aware  of  tools  and  technologies  that  may  be  applicable 
to  the  support  process  of  that  organization  in  order  to  provide  better  software  support. 

The  most  immediate  improvement  for  this  hypothetical  organization  will  come  from 
analyzing  the  “no”  responses  of  those  questions  that  fall  into  the  Level  2  categories  of 
software  support  process  and  personnel  and  the  Level  1  and  2  tools  and  technology  category. 
This  provides  specific  factors  that  will  help  the  organization  mature  in  a  smooth  fashion. 
It  would  not  make  sense  for  this  organization  to  concentrate  on  Level  5  questions  for  any 
category  without  having  considered  questions  contained  in  Levels  3  and  4  first. 
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A  Glossary  of  Terms 


Acceptance  Review  A  review  of  a  software  product  by  developers  and  maintainers  to 
determine  if  the  product  satisfies  ali  originally  specified  requirements. 

Acceptance  Test  Testing  led  by  the  client  or  QA  group  to  determine  whether  the  product 
satisfies  its  specifications  as  claimed  by  the  developer. [Sch90] 

Application  System  same  as  Information  System 

Availability  A  measure  of  the  degree  to  which  an  item  is  in  an  operable  and  committable 
state  at  the  start  of  a  mission  when  the  mission  is  called  for  at  a  random  point  in 
time.[Dep82] 

Benchmark  Testing  Evaluation  of  the  system  performance  against  quantitative  requirements. [Sch90] 

Change  Request  Review  Board  An  authority  responsible  for  evaluating  and  approving 
requests  for  changes  to  a  software  product. 

Cohesion  A  measure  of  the  degree  of  the  functional  relav-edness  within  program  units. 

[Som89] 

Complexity  A  characteristic  of  the  software  interface  which  influences  the  resources  an¬ 
other  system  will  expend  or  commit  while  interfacing  with  the  software.  [CDS86] 

Configuration  Management  The  process  of  identifying  and  defining  the  configuration 
items  (hardware/software  units)  in  a  system,  controlling  the  release  and  change  of 
these  items  throughout  the  system  life  cycle,  recording  and  reporting  the  status  of 
configuration  items  and  change  requests,  and  verifying  the  completeness  and  correct¬ 
ness  of  configuration  items.  [IEE83] 

Consistency  The  extent  to  which  uniform  design  techniques  and  notation  are  used.  [War87] 

Coupling  A  measure  of  the  strength  of  interconnections  (dependencies)  between  program 
units.  [Som89] 

Error  Human  action  that  results  in  software  containing  a  fault.  Examples  include  omis¬ 
sion  or  misinterpretation  of  user  requirements  in  a  software  specification,  incorrect 
translation  or  omission  of  a  requirement  in  the  design  specification.  [IEE83] 

Failure  A  departure  of  program  operation  from  program  requirements.[IEE83] 

Failure  Rate  The  number  of  failures  of  an  item  per  measure-of-life  unit.[Dep82] 

Fault  A  manifestation  of  an  error  in  software.  A  fault,  if  encountered,  may  cause  a  failure. 
Synonymous  with  bug. 

Fourth  Generation  Language  (4GL)  A  computer  programming  language  that  provides 
abstractions  of  data  and/or  procedural  specifications  and  is  usually  suited  for  a  par¬ 
ticular  application  domain. 
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Integration  Testing  Verify  that  the  modules  of  the  system  combine  correctly  in  order  to 
achieve  a  product  that  meets  its  specifications.  (Sch90] 

IS  (Information  Systems)  Organization  An  organied  collection  of  procedures,  person¬ 
nel,  and  resources  dedicated  to  support  a  portfolio  of  information  systems. 

Lines  of  Code  Lines  of  source  code,  not  including  comments.. 

Maintainability  The  probability  that  an  item  will  be  retained  in,  or  restored  to,  a  specified 
condition  within  a  given  period  if  prescribed  procedures  and  resources  are  u8ed.[Dep82] 

Maintenance  All  actions  required  to  retain  an  item  in,  or  restore  it  to,  a  specified  condition. [Dep82] 

Maintenance  Audit  An  organized  review  of  the  maintenance  organization. 

Maintenance  Escort  Participation  of  the  softwaje  maintainer  in  software  system  devel¬ 
opment. 

Man/Machine  Interface  The  software  that  supports  the  interaction  between  the  user 
tuid  the  system. 

Measure  A  high-level  unit  of  specification  which  characterizes,  evaluates,  or  predicts  var¬ 
ious  aspects  of  software  life  ’.ycle  processes  and  products. 

Metric  A  measurable  indication  of  some  aspect  of  a  system.  [DeM82]  A  quantification  of 
a  specific  feature  of  the  software  life  cycle  process  or  software  product. 

Modularity  A  characteristic  of  software  such  that  it  is  well-structured,  highly  cohesive, 
and  minimally  coupled.  [War87] 

New  Systems  Development  The  development  of  a  system  which  has  never  been  fielded. 

Object  Oriented  Design  Designing  a  system  in  terms  of  abstract  data  types  where  the 
objects  are  instantiations  of  the  data  types  and  new  data  types  can  be  defines  as 
extensions  of  previously  defined  types. 

Regression  Testing  Testing  the  system  against  previous  test  cases  to  ensure  that  the 
functionality  of  the  system  has  not  been  compromised  by  recent  changes  to  the  system. 

[Sch90] 

Reliability  The  probability  that  an  item  will  perform  its  intended  function  for  a  specified 
interval  under  stated  conditions.[Dep82] 

Self- Descriptiveness  A  characteristic  of  software  that  enables  the  understanding  of  im¬ 
plementation  of  software  functions.  [War87] 

Support  Staff  The  personnel  tasked  with  maintaining  an  information  system. 

Supportability  A  measure  of  the  adequacy  of  products,  resources,  and  procedures  to 
facilitate  the  support  activities  of  modifying  and  installing  software,  establishing  an 
operational  software  baseline,  and  meeting  user  requirements.  [PTH87] 
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Testability  The  extent  to  which  software  facilitates  both  the  establishment  of  test  criteria 
and  the  evaluation  of  the  software  with  respect  to  those  criteria.  [IEE83] 

Throw-away  prototyping  Creating  a  prototype  as  part  of  system  design  and  then  "throw¬ 
ing  away”  the  prototype  and  implementing  the  system  "from  scratch”  not  using  any 
of  the  source  code  from  the  prototype. 

Top-down  design  Designing  the  system  by  recursively  breaking  the  system  down  into 
smaller  components. 

Unit  Testing  Testing  of  individual  portions  of  the  system. 
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B  List  of  Acronyms 

AIRMICS  U.S.  Army  Institute  for  Research  in  Management  Information,  Communicar 
tions,  and  Computer  Science 

AMC  Army  Materiel  Command 

CCB  Change  Control  Board  I 

COE  Army  Corps  of  Engineers 
FORSCOM  Forces  Command 

HSC  Army  Health  Services  Command  ^ 

IS  Information  System 

ISC  Army  Information  Systems  Command 

LOC  Lines  of  Code 
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C  Organization  Assessment  Questionnaire 

This  appendix  contains  a  12  page  questionnaire  (not  including  the  questionnaire  cover  page) 
for  gathering  organizational  assessment  data.  The  questionnaire  should  be  photocopied  and 
distributed  to  selected  respondents. 


» 


11 


I 


SOFTWARE  SUPPORTABILITY  QUALITATIVE 
ASSESSMENT  METHODOLOGY 


ORGANIZATION  ASSESSMENT 
QUESTIONNAIRE 


Sottware  Supportability  Qualitative  Assessment  Metltedolo^v 
Organization  Assessment  Questionnaire 


ORGANIZATIONAL  ISSUES 


1.1  Organizational  Structure: 

YES 

1.1.1.  Are  departments /groups  in  the  software  organization  structured 
on  the  basis  of  life  cvcle  phase  (separate  Development  and 
Maintenance  groups)?  _ 


NO 


1.1.2.  Are  departments/groups  in  the  software  support  organization 
structured  on  the  basis  of  application  skill  (financial  applications 
payroll  applicadorts  etc.)? 

1.1.3.  Are  departments/groups  in  the  software  organization  structured 
on  the  basis  of  technical  skills  (separate  groups  for  systems  analysts, 
programmers)? 

1.1.4.  Do  the  same  software  personnel  perform  both  development  and 
maintenance  functions? 

1.1.5.  Are  there  specific  measures  currently  used  for  determining  the 
effectiveness  of  your  software  maintenance  organization? 

1.1.6.  For  each  project  involving  software  maintenance,  is  there  a 
designated  software  manager? 

1.1.7.  If  so,  does  this  software  manager  report  to  an  overall  project 
manager? 

1.1.8.  Does  the  Software  Maintenance  Quality  .Assurance  function 
have  a  management  reporting  channel  separate  from  the  software 
project  management? 

1.1.9.  Is  there  a  designated  individual  or  team  responsible  for  the 
control  of  software  interfaces  (refer  to  question  in  tools  section  3.2.10)? 

1.1.10.  Is  software  system  engineering  represented  on  the  software 
maintenance  team? 


1.1.11.  Is  there  a  software  configuration  control  function  for  each 
software  maintenance  project’ 
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Sv''!(ware  Supportabilitv  Quaiitatne  Assessment 

Organization  Assessment  Questionnaire _ 


YES  \0 

1.1.12.  Is  there  a  formal  procedure  or  plan  for  co-ordinating  tasks 

within  the  support  staff?  _  _ 

1.1.13.  Is  there  a  specific  group  that  determines  software  maintenance 

procedures?  _  _ 

1.1.14.  Aie  there  formal  job  descriptions  available  for  each  member  of 

the  staff?  _  _ 


1.2  Portfolio  Characteristics  and  Management: 

1.2.1.  Are  there  fewer  than  five  different  source  languages  that  you 
support  in  the  application  portfolio? 

1.2.2.  Are  profiles  maintained  of  sizes  of  applications  in  the  application 
portfolio? 

1.2.3.  Are  profiles  maintained  of  the  resources  expended  per 
application  in  the  application  portfolio? 

1.3  Physical  Facilities: 

1.3.1.  Does  each  maintenance  programmer  have  adequate  access  to 
appropriate  computing  facilities? 

1.3. 2.  Can  you  emulate  the  user  hardware  and  software  configurations 
for  each  application  in  your  portfolio? 

1.4  Budgetary  Control: 

1.4.1.  Is  at  least  30%  of  the  maintenance  budget  spent  on  improving 
maintenance  Quality? 

1.4.2.  Is  there  a  separate  budget  for  maintenance  and  development 
projects? 

1.4  3.  Is  there  a  separate  budget  for  each  application  that  is  supported  in 
the  support  portfolio? 
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Software  Support.ibility  A^sossn'cnl  NUtl'-iviologv 

_ Organization  Assossnieiu  Questionnaire  _ 


ms  mj 

1.5  Effectiveness  of  the  organization  as  a  whole: 

1.51.  Is  the  organizational  effectiveness  predominantlv  deterniined  by 
the  size  of  the  change  request  backlog?  _ : 

1.5.2.  Is  there  a  mechanism  for  user  evaluation  of  the  support 

organization's  effectiveness?  _  _ 

1.5.3.  Would  you  perceive  the  quality  and  the  effectiveness  of  the 
support  function  that  you  provide  to  be: 

(i)  poor?  _  _ 

(i)  fair?  _  _ 

(iii)  good?  _ 

(iv)  excellent?  _  _ 

Relationship  with  User  Organization: 

1.6.1.  Do  you  consider  90%  of  the  user  population  of  your  application 

portfolio  to  be  'computer  literate  ?  _  _ 

1.6.2.  Do  you  keep  a  record  of  all  communications  'net ween  your 

organization  and  user  organizations?  _  _ 

1.6.3.  Are  over  50%  of  the  communications  between  your  support 

organization  and  the  users  initiated  by  your  organization?  _  _ 

1.6.4.  Do  you  have  a  newsletter  or  other  regular  vehicle  for 

communicating  with  your  user  organizations?  _ 

1.6.5.  Is  there  a  formal  mechanism  for  discussing/negotiating  change 

requests  initiated  and  their  impacts  with  the  users?  _  _ 

1.6.6.  Do  the  users  consider  the  quality  and  effectiveness  of  the  support 

function  as  good?  _ 

1.6.7.  Would  you  say  that  the  current  application  staff  communicates 
directly  with  the  users  of  the  applications: 

(i)  daily?  _ _ 

(it)  not  daily  but  atleast  weekly?  _  __ 

(iii)  not  weekly  but  atleast  monihlv? 


Sottware  Supportabiiiiv  Quahtativo  Assessment  Methodoiogy 
_ Organizat:on  Assessment  Questionnaire 
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1.6  8.  Would  you  concur  that  the  users  reasonably  understand  the 

application  system  they  use  (or  are  they  experienced  in  the 
applications  they  use)?  _ 

1.6.9.  Do  you  think  the  user  expectations  (in  terms  of  change  requests 
and  enhancements)  are  realistic?  _ 

1.7  Relationship  with  the  Development  Organization: 

1.7.1.  Is  there  a  formal  channel  for  communication  between  the 
maintenance  personnel  and  the  developers  of  the  software  with 
regard  to  maintainability  of  the  software: 

(i)  during  the  development  stage  of  the  software?  _ 

(ii)  after  the  installation  of  the  software?  _ 


SOFTWARE  SUPPORT  PROCESS  AND  ITS  .MANAGEMENT 
2.1  Standards  and  Procedures: 

2.1.1.  On  each  supported  software  system,  does  your  organization  have 
a  set  of  standardized  and  documented  procedures  to  follow  in: 

(i)  modifying  system  code? 

(ii)  testing  system  code? 

(iii)  using  specific  tools  and  techniques  for  using  them? 

2.1.2.  Are  formal  procedures  used  for 

(i)  estimating  sizes/extent  of  changes  to  systems? 

(ii)  estimating  software  maintenance  cost  (over  a  given  period  of 
time)? 

(iii)  tracking  the  size  of  software  system(s)  being  maintained? 

2.1.3.  Is  a  mechanism  used  for  ensuring  that  the  software  maintenance 
team  becomes  familiar  with  the  system  being  maintained? 

2.1.4.  Are  standards  applied  to  software  in  a  maintenance  project? 

2.1.5.  Are  standards  applied  to  the  preparation  of  unit  test  cases? 

2.1.6.  Are  re-design  review  standards  applied? 
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2.1.7  Are  there  formal  organizational  proct\iurcf  tor  prioritizing  the 
incoming  change  requests?  _  _ 

2.1.8.  Are  there  formal  organizational  pnocedures  tor  prioritizing  the 
maintenance  work  with  competing  demands  tor  time?  _  _ 

2.2  Process  Metrics: 

2.2.1.  Are  records  of  manpower  expenditures  maintained  for  actual  vs. 

planned  software  support?  _  _ 

2.2.2  Are  records  of: 

(i)  planned  vs.  actual  maintenance  completion  dates  m.aintained?  __  _ 

(ii)  planned  vs.  actual  testing  completion  durations  maintained?  '  _ 

(iii)  software  units  maintained  (over  a  given  period  of  time)?  __  _ 

2.2.3.  Are  statistics  on: 

(i)  software  code  and  test  errors  gathered?  ■  _j_  •  _ 

(ii)  software  design  error  gathered?  _  _ 

2.2.4.  Are  the  following  tracked  to  closure: 

(i)  Action  items  resulting  from  design/ maintenance  reviews  _  _ 

(ii)  Action  items  resulting  from  code  reviews  _  _ 

(iii)  Software  trouble  reports  resulting  from  testing  _  _ 

2.2.5.  Are  there  formal  organizational  procedures  for: 

(i)  measuring  the  throughput/effectiveness  of  the  support  function?  _ _  _ 

(ii)  measuring  the  support  staff  performance?  _  _ 

2.3  Management  of  the  Support  Process 

2.3.1.  Is  a  mechanism  used  for  measuring  characteristics  of  the 

application  portfolio  (i.e  complexity,  size,  age,  technology)?  _  _ 

2.3.2.  Is  a  mechanism  used  for  measuring  and  monitoring  the  costs  for 

each  support  task/  project?  _  _ 

2.3.3.  Is  a  mechanism  used  for  measuring  and  monitoring  the 

workload  for  each  support  task/  project?  _ 
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2.3  4.  Is  a  mechanism  used  for  monitoring  and  analyzing  the  nature  of 
each  support  task  perlormed  (nature  categorized  as  perfective, 
adaptive  and  corrective)?  ^  _  _ 

2.3.5.  Is  there  a  formal  procedure  for  maintaining  support  work 

history?  _  _ 

2.3.6.  Is  there  a  mechanism  for  tracking  the  source  of  maintenance 

requests?  _  _ 

2.3.7.  Is  there  a  mechanism  to  ensure  that  programmers  meet 

scheduled  commitments?  _  _ 

2.3.8.  Does  senior  management  have  a  mechanism  for  the  regular 

revie^v  of  the  status  of  software  system/units  maintained?  _  _ 

2.3.9.  Are  the  error  causes  reviewed  to  determine  the  process  changes 

required  to  prevent  them?  _  _ 

2.3.10.  Do  the  technical  interchanges  include  information  regarding . 

size,  complexity,  number  of  errors,  etc.?  _  __ 

2.3.11.  Do  you  have  i  formal  procedure /sche&ale  for: 

(i)  regular  technical  interchanges  with  the  user?  _  _ 

(ii)  regular  technical  interchanges  with  the  developer/designer  of  the 

system  being  maintained?  _  _ 

2.3.12.  Is  the  error  data  from  code  reviews  and  tests  analyzed  to 

determine  the  likely  distribution  of  the  errors  remaining?  _  _ 

2.3.13.  Do  you  have  a  formal  procedure  for: 

(i)  assessing  the  support  process  and  implementing  the  recommended 

improvements  (i.e.,  conducting  internal  maintenance  reviews)?  _  _ 

(ii)  controlling  changes  to  code  (software  requirements)?  _  _ 

(iii)  deciding  when  to  insert  new  technology  into  the  support  process?  _  _ 

(iv)  managing  and  supporting  the  introduction  of  new  technologies?  _  _ 

(v)  recording  software  unit/system  maintenance  progress?  _  _ 

(vi)  conducting  planned  maintenance  on  application  software?  _  _ 

(vii)  conducting  periodic  maintenance  audits?  _  _ 

2.3  14.  Do  vou  have  a  formal  change  request  review  process?  _  _ 
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YES 

2.3.15.  Is  there  a  formal  impact  analysis  procedure  adopted  in  you 

organization?  _ 

2.3.16.  Do  you  share  the  results  oi  the  impact  analysis  of  the  change 

requested  with  the  user?  __ 

2.3.17.  Are  the  following  organizational  technioues  used: 

(i)  change  request  procedure?  _ 

(ii)  scheduled  maintenance?  _ 

(iii)  formal  retest  procedures?  _ 

(iv)  change  request  review  board?  _ 

(v)  chargeback  for  operations  and  maintenance?  _ 

(vi)  periodic  maintenance  audit?  _ 

2.3.18.  Are  the  following  work  methods  used: 

(i)  top  -  down  design?  _ 

(ii)  structured  programming?  •  _ 

(iii)  structured  walk  through?  _ 

(iv)  checkpoint  review?  _ 

(v)  library  of  previous  problems  and  application  functions?  _ 

(vi)  test  data  generator?  _ 

(vii)  benchmark  testing  ?  _ 

(viii)  programmer  workbench?  _ 

2.3.19  Are  the  following  Monitoring  techniques  used: 

(i)  McCabe’s  Cyclomatic  comple.xity  number?  _ 

(ii)  McClure’s  control  flow  metric?  _ 

viii)  Henry  and  Kafara’s  information  rlow  metric?  _ 


TOOLS  AND  TECHNOLOGY 
3.1  Technology  Management: 

3.1.1.  Is  a  mechanism  used  for  maintaining  awareness  of  the  state-of- 
the-art  in  software  engineering  technology? 

3.1.2.  Is  a  mechanism  used  for  evaluating  technologies  used  by  the 
organization  versus  those  e.xternally  available? 
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3.1.3.  Is  a  mechanism  used  for  deciding  when  to  insert  new  technolocv 
into  the  maintenance  process? 

3.1.4.  Is  a  mechanism  used  for  managing  and  supporting  the 
introduction  of  new  technologies? 

3.1.5.  Is  a  mechanism  used  for  identifying  and  replacing  obsolete 
technologies? 

3.2  Tools  and  Techology  used  in  the  Softw’are  Support  process: 

3.2.1.  Are  manual  testing  techniques  used  to  e.xamine  the  computer 
program  in  order  to  detect  particular  characteristics  of  the  code? 

3.2.2.  Are  automated  testing  techniques  used  to  examine  the  computer 
program  in  order  to  detect  particular  characteristics  of  the  code? 

3.2.3.  Which  of  the  following  automated  testing  tools  are  used: 

(i)  Error  detection  aids  -  for  detecting  and  removing  coding  error? 

(ii)  Anomaly  detection  aids  -  for  detecting  discrepancies  in  the  form 
and  syntax  of  the  code? 

(iii)  Structural  analysis  -  automated  techniques  for  characterizing  the 
logical,  data  and  control  structures  of  computer  programs? 

(iv)  Test  data  generation  -  specially  developed  to  satisfy  individual  and 
unique  project  and  test  requirements? 

(v)  Program  metrics  -  for  classifying  and  estimating  error  types, 
performing  cost  estimations  and  establishing  program  complexity 
measures? 

3.2.4.  Are  dynamic  testing  techniques  used  (which  require  program 
execution  for  analysis)? 

3.2.5.  Which  of  the  following  dynamic  testing  tools  are  used: 

(i)  Error  detection  aids  -  which  involves  isolating,  detecting  and 
removing  errors  from  program  code  while  executing  the  program? 

(ii)  Structural  analysis  -  as  in  Automated  testing  techniques,  but 
requiring  program  execution  for  logic,  data  and  code  analysis? 

(iii)  Functional  analysis  -  black  box  testing,  input/output  driven,  based 
on  program  performance  requirements  and  functionality? 

(iv)  Performance  monitors  -  automated  techniques  to  collect 
performance  data  on  computer  program  execution  characteristics? 
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YUS  NO 

3.2.6.  Are  test  management  techniques,  which  aid  in  the 

docunnentation,  control  and  conduct  ot  testing  used?  _ 

3.2.7.  Are  formal  techniques  involving  rigorous  symbolic,  algebraic  or 

mathematical  manipulation  of  computer  programs  used  for  formally- 
verifying  computational  properties  and  performance  goals?  _  _ 

Tools/tecKnology  issues: 

3.3.1.  Are  the  tools  used  by  development  compatible  with  those  used 

by  maintenance?  _  _ 

3.3.2.  Do  most  tools  support  a  majority  of  the  languages  being  used?  _  _ 

3.3.3.  Do  most  tools  support  a  majority  of  the  systems  being  used?  ■  _ 

3.3.4.  Is  a  mechanism  used  for  ensuring  smooth  transition  of  a  system 

from  development  to  maintenance?  __ 

3.3.5.  Is  any  effort  being  made  for  developing  an  integrated  set  of  tools 

to  support  each  phase  of  the  software  lifecycle  (development  through 
maintenance)?  _ 

3.3.6.  Is  there  a  formal  process  for  reporting  the  size  of  errors,  number 

of  changes  made,  and  complexity  of  errors  to  the  development  group?  _  _ 

Documentation  tools: 

3.4.  Are  the  following  documentation  tools  used: 

(i)  User  manual?  _ 

(ii)  Data  dictionary?  _ 

(iii)  Data  flow  diagram?  _ 

(iv)  Operations  error  history?  _ 

(v)  System  maintenance  journal?  _ 

(vi)  Pseudo-code?  _ 

(vii)  HIPO  diagram?  _ 

(viii)  Data  model  diagram?  _ 

(ix)  Test  history?  _ 

(x)  Automated  code  analyzer?  _ 

(xi)  Warmer  diagram?  _ 

(xii)  Jackson  diagram?  _ 


f 

I, 


o 


Souu-are  5urrortab:luy  Quahta-.ivo  A>sos<mcnt  Mw^thoJo’o.uv 
Organization  A5?c>sn^icnt  Qu^stionnairt’ 


PERSONNEL  MANAGEMENT 


M:S  NO 


4.1.  Training: 

4.1.1.  Do  software  personnel  simultaneouslv  work  on  development 
and  maintenance  projects? 

4.1.2.  Is  there  a  meckjnisyn  for  measuring  and  improving  support 
personnel  skills? 

4.1.3.  Is  there  a  required  training  program  for  all  newlv  appointed 
support  managers  designed  to  familiarize  them  with  software  support 
project  management? 

4.1.4.  Is  there  a  required  software  engineering  training  program  for; 

(i)  software  maintenance  personnel? 

(ii)  first  line  supervisors  of  soft^vare  support? 

4.1.5.  Is  there  a  formal  training  program  required  for: 

(i)  the  support  team  leaders? 

(ii)  software  change  specification  writers? 

(iii)  software  test  specification  writers? 

(iv)  software  documentation  writers? 

4.1.6.  Is  there  a  formal  user  training  program  offered  for  each  software 
product  that  is  being  supported? 

4.1.7.  For  those  employees  with  at  least  3  years  of  e.xperience,  have  at 
least  one  third  of  them  had  some  formal  re-currency  training  in  the 
last  3  years? 

4.2  Experience: 

4.2.1.  Are  software  personnel  rotated  between  different  departments/ 
groups? 

4.2.2.  Are  there  several  occasions  \vhen  support  personnel  are  required 
to  work  overtime? 

4.2.3.  Is  there  a  mechanism  to  ensure  that  there  are  sufficient 
knowledgeable  and  trained  support  personnel  for  every  application? 
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NO 

4.2.4.  Do  all  the  soff-eare  support  managers  have  at  least: 

(i)  5  years  of  exp>erience  with  the  support  organization?  _  _ 

(ii)  10  years  of  experience  vvith  the  support  organization?  _  _ 

4  2.5.  Do  all  the  software  support  personnel  have  at  least; 

(i)  some  formal  technical  training  m  :he  area  of  software  support?  _  _ 

(ii)  a  2  year  degree  or  equivalent  in  a  computer  related  discipline?  _  _ 

(iii)  a  4  year  degree  or  equivalent?  _  _ 

(iv)  a  4  year  degree  in  computer  science  or  software  engineering?  _  _ 

(v)  3  years  experience  in  software  support?  _  _ 

4.2.6.  Do  at  least  two  thirds  of  the  software  support  personnel  have: 

(i)  at  least  6  years  experience  in  software  support?  _  _ 

(ii)  at  least  1  years  experience  with  all  the  tools  and  applications?  _ 

(iii)  at  least  1  years  experience  with  all  the  languages  used?  _ 

4.3  Turnover  Rate: 

4.3.1.  Is  the  total  number  of  employees  that  have  left  the  support 

organization  for  any  reason  in  the  last  3  years: 

(i)  less  than  10  percent  of  the  current  number  of  support  staff?  _ 

(ii)  less  than  30  percent  of  the  current  number  of  support  staff?  _ 

(iii)  less  than  50  percent  of  the  current  number  of  support  staff?  _ 

4.4  Recruitment/MotivationyEvaluation: 

4.4.1.  Is  there  a  formal  procedure  for: 

(i)  career  planning?  _ 

(ii)  support  personnel  selection?  _ 

4.4.2.  Is  there  a  mechanism  for  measuring  and  improving: 

(i)  support  personnel  motivation?  _ 

(ii)  support  personnel  productivity?  _  _ 

4.43.  Do  you  consider  the  motivation  level  of  staff  as  reasonably  high?  _  _ 

4.4.4.  Do  all  personnel  know  exactly  what  their  functions/duiies  are?  _ 

4.4.5.  Would  you  term  the  relationship  between  staff  and  managers  as: 

(i)  good?  _ 

(ii)  excellent? 
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D  Matri^c 


J’his  appendix  rontains  an  illustration  of  the  organizational  assessment  maturity  matrix. 
Maturity  levels  are  dejiirted  on  the  horizontal  axis,  and  organizational  support  factors  are 
shown  on  the  vortical  axis.  Within  each  matrix  cell  is  a  listing  of  organizational  assessment 
questions  corresponding  to  a  given  level  of  maturity  and  organizrational  support  factor. 


Matrix 


E  Assessment  Recording  Form 


This  appendix  contains  the  form  to  record  the  answers  to  organizational  assessment  ques¬ 
tions  (see  Appendix  (■).  One  copy  of  this  form  slioulil  be  be  provided  al*>ng  with  the  <|nes- 
tionnaire  in  Appendix  C  to  each  respondent. 


S()llv..iro  Support  Or v;.uii/.itiou  Assi'ssmi'ut 


\ns'.vors  to  ‘I'csC  'Uouli.!  ;u;it.i.'t  '■taruiurv.i  pr.icoco 


Question 

Number 

iZ6 

1  6.7(i) 

1  6  7(ii) 

1 .6  7(iii) 

TTs 

1  6.9 
1.7.1(i) 

2.1.1(i) 
2.1. Kii) 
2.1.1(iii) 
2.1.20) 
2.1.20i) 
~2.1  20ii) 

2.1.3 

2.1.4 

2.1.5 

2.1.6 

2.1.7 

2.1.8 

2.2.1 

2.2.20) 

2.2.2(ii) 

2.2.20ii) 

2.2.3(i) 

2.2.30i) 

2.2.40) 

2  2.40i) 
2.2.4(iii) 
2.2.50) 
2.2.50i) 

2  3.1 

2.3.2 

2.3.3 


Level 

4 


Comments 


Select 
Ans^vcr 
Y  I  N 


2 
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Answers  to  these  questions  should  ror.oct  standard  uruam 


Question 


saliona! 


Number 


Level 


3 


Comments 


practice 


Select 

Answer 


N 


2.3. 


2.3.8 


2.3.9 


2.3.10 


2.3.11(1) 


2.3.11(ii) 


2.3.12 


2.3.13(1) 


2.3.13(11) 


2.3.13(111) 


2.3.13(lv) 


2.3.13(v) 


2.3.13(vi) 


2.3.13(vll) 


2.3.14 


2.3.15 


2.3.16 


2.3.17(1) 


2.3.17(11) 


2.3.17(111) 


2.3.17(lv) 


2.3.17(v) 


2.3.17(vl) 


2.3.17(vll) 


2.3.18(1) 


2.3.18(11) 


2.3.18(111) 


2.3.18(lv) 


2.2.18(v) 


2.3.18(vl) 


2.3.18(vll) 


2.3.18(vlll) 


3 
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Answers  to  these  O[i:esl:ons  ■'hould  rotloct  'standard  organi/alu'na!  pr.‘ 


Question 

Number 


2.3.19(i) 


2.3.19(ii) 


2.3.19(iii) 


3.1.2 


3.1.3 


3.1.4 


3.1.5 


3.3.6 

3 

3.4(i) 

1 

3.4(ii) 

1 

3.4(iii) 

2 

3.4(iv) 

3.4(v) 

3.4(vi) 


Soitwjre  Support  Organi^Jlion  Assessment 


Answers  to  these  ouesUons  “-hould  leilect  standard  urgam/, 


Question 


;i:onal  praetice 


Level 


Number 


3.4(vii) 


3.4(viii) 


3.4(ix) 


3.4(x) 


3.4(xi) 


3.4(xii) 


4.1.1 


4.1.2 


4.1.3 


4.1.4(i) 


4.1.4(ii) 


4.1. 5(i) 


4.1.5(ii) 


4.1.5(iii) 


4.1.5(iv) 


4.1.6 


4.1.7 


4.2.1 


4.2.2 


4.2.4(i) 


4.2.4{ii) 


4.2.5(i) 


4.2.5(ii) 


4.2.5(iii) 


4.2.5(iv) 


4.2.5(v) 


4.2.6(i) 


4.2.6(ii) 


4.2.6(iii) 


4.3.1(i) 


4.3.1 


4.3.1(iii) 


4.4.1(i) 


4.4.1(ii) 
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F  User’s  Guide 


This  appendix  rontaiiis  a  forms  illustrating  an  ‘automated  questionnaire'  tool  that  has 
been  developed  to  facilitate  the  organisational  assessment  questionnaire  completion  process. 
The  appendix  also  contains  an  additional  "reference  questionnaire"  illustrating,  for  each 
question,  the  maturity  level  corresponding  to  the  question. 


Software  Support  Organization  Assessment 

Questionnaire 


User's  Guide 


Users  Guide 


The  automated  Software  Support  Organization  A>^es?ment  has 

designed  to  facilitate  the  user  in  answering  the  questionnaire  b\  providing  a 
computer  guided  walkthrough  with  user-friendly  prompts  At  the  end  of  a 
consultation  session,  the  automated  questionnaire  also  calculates  the  Maturity  Level 
of  the  organization  and  indicates  areas  for  improvement 

The  automated  questionnaire  has  been  developed  using  \’P-E\pert ,  a  rule-based 
expert  system  development  tool  for  the  IBM  PC.  The  automated  questionnaire  is 
very  easy  to  use  and  requires  almost  no  prior  PC  knowledge 

The  following  figures  give  a  step  by  step  walkthrough  of  a  consultation  session; 


USERS  GUIDE 


STEP  1 

Insert  the  diskette  into  the  A  Drive. 
At  the  DOS  prompt  type  VPX 
A:\>  VPX 

The  first  screen  will  now  appear. 


STEP  2 


VP  -  EXPERT 
Version  2.1 


1  Help  2  induce  3  Edit  4  Consult 


[FACTS] 


5  Tree  6  Filename  7  Path  8  Quit 


STEP3 


JP  -  EXPERT 
Version  2.1 


1  Help  2  Induce  3  Edit  4  Consult  5  Tree  6  Filename  7  Path  8  Quit 


STEP  4 


[KBS.QS] 


Loading  file... 


1  Help  2  induce  3  Edit  4  Consult  5  Tree  6  Filename  7  Path  8  Quit 


STEP  5 


This  questionnaire  is  designed  to  help  determine  the 
software  support  capability  of  an  organization. 


press  any  key  to  begin  the  consultation 


■- 

V  'X?  •  t  .  f,*  - 

^  S  '  \  '  '  s 

/  •  .  ./  V. 

'  y 
'yy' 

s’" 

.■^1' 

'  s'  . 

, ;  ■ 

.  s' 

S' 

STEP 


SOFTWARE  SUPPORT  ORGANIZATION  ASSESSMENT  QUESTIONNAIRE 


DEVELOPED  BY  CIMR 


All  answers  should  reflect  current  organizational  practice 


<Press  any  key  to  Continue> 


Place  the  cursor  on  the  selected  answer  by  using  the  arrow  keys. 
Press  "Enter"  or  "Return"., 

Another  Question  Screen  similar  to  the  one  shown  above  will  appear. 

Continue  to  answer  questions  from  the  Question  Screens  as  described 
above. 


» 


Questions  with  sub-parts  will  scroll  through  on  the  screen.  New 

questions  will  appear  at  the  top  of  similar  Question  Screens.  ^ 

When  all  questions  in  the  questionnaire  have  been  answered,  the  system 
will  compute  the  organizational  level.  A  flashing  screen  asking  the 
user  to  wait  will  appear.  After  about  10  seconds,  the  final  screen  as 
shown  in  STEP  8  will  appear.  i 


» 
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